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Abstract. Modern ontology debugging methods allow efficient identification and 
localization of faulty axioms defined by a user while developing an ontology. The 
ontology development process in this case is characterized by rather frequent and 
regular calls to a reasoner resulting in an early user awareness of modeling errors. 
In such a scenario an ontology usually includes only a small number of conflict 
sets, i.e. sets of axioms preserving the faults. This property allows efficient use of 
standard model-based diagnosis techniques based on the application of hitting set 
algorithms to a number of given conflict sets. However, in many use cases such 
as ontology alignment the ontologies might include many more conflict sets than 
in usual ontology development settings, thus making precomputation of conflict 
sets and consequently ontology diagnosis infeasible. In this paper we suggest a 
debugging approach based on a direct computation of diagnoses that omits cal- 
culation of conflict sets. Embedded in an ontology debugger, the proposed algo- 
rithm is able to identify diagnoses for an ontology which includes a large number 
of faults and for which application of standard diagnosis methods fails. The eval- 
uation results show that the approach is practicable and is able to identify a fault 
in adequate time. 



1 Introduction 

Ontology development and maintenance relies on an ability of users to express their 
knowledge in form of logical axioms. However, the knowledge acquisition process 
might be problematic since a user can make a mistake in an axiom being modified or a 
correctly specified axiom can trigger a hidden bug in an ontology. These bugs might be 
of different nature and are caused by violation of requirements such as consistency of 
an ontology, satisfiability of classes, presence or absence of some entailments. In such 
scenarios as ontology matching the complexity of faults might be very high because 
multiple disagreements between ontological definitions and/or modeling problems an 
be triggered by aliments at once. 

Ontology debugging tools [8 2 4] can simplify the development process by allowing 
their users specification of requirements to the intended (target) ontology. If some of the 
requirements are broken, i.e. an ontology O is faulty, the debugging tool can compute a 
set of axioms DCO called diagnosis. An expert should remove or modify at least all 
axioms of a diagnosis in order to be able to formulate the target ontology Ot - Neverthe- 
less, in real-world scenarios debugging tools can return a set of alternative diagnoses D, 
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since it is quite hard for a user to specify such set of requirements that allows formula- 
tion of the target ontology O t only. Consequently, the user has to differentiate between 
multiple diagnoses T> € D in order to find the target diagnosis T> t , which application 
allows formulation of the intended ontology. Diagnosis discrimination methods B9I18I 
allow their users to reduce the number of diagnoses to be considered. The first ap- 
proach presented in [9] uses a number of heuristics that rank diagnoses depending on 
the structure of axioms axi € T>, usage in test cases, provenance information, etc. Only 
diagnoses with the highest ranks are returned to the user. A more sophisticated approach 
suggested in [18] identifies the target diagnosis by asking an oracle, like an expert or 
information extraction system, a sequence of questions: whether some axiom is entailed 
by the target ontology or not. Given an answer the algorithm removes all diagnoses that 
are inconsistent with it. Furthermore, the query is used to create an additional test case, 
which allows the search algorithm to prune the search space and reduce the number of 
diagnoses to be computed. Moreover, in order to speed up the computations the method 
approximates the set of all diagnoses with a set of n leading diagnoses, i.e. the n best 
diagnoses with respect to a given measure. 

All the approaches listed above follow the standard model-based diagnosis ap- 
proach lfl6l and compute diagnoses using minimal conflict sets CS, i.e. irreducible 
sets of axioms axi 6 O that preserve violation of at least one requirement. The com- 
putation of the conflict sets can be done within a polynomial number of calls to the 
reasoner, e.g. by QuickXPlain algorithm |7). To identify a diagnosis of cardinality 
| V | = m the hitting set algorithm suggested in 1 16 1 requires computation of m minimal 
conflict sets. In the use cases when an ontology is generated by an ontology learning or 
matching system the number of minimal conflict sets m can be large, thus making the 
ontology debugging practically infeasible. 

In this paper we present two algorithms Inv-HS-Tree and Inv-QuickXPlain, 
which inverse the standard model-based approach to ontology debugging and compute 
diagnoses directly, rather than by means of minimal conflict sets. Thus, given some 
predefined number of leading diagnoses n the breadth-first search algorithm Inv-HS- 
Tree executes the direct diagnosis algorithm Inv-QuickXPlain exactly n times. 
This property allows the new search approach to perform well when applied to ontolo- 
gies with large number of conflicts. The evaluation shows that the direct computation 
of diagnoses allows to apply ontology debugging in the scenarios suggesting diagnosis 
of generated ontologies. The system based on Inv-QuickXPlain is able to compute 
diagnoses and identify the target one in the cases when common model-based diagnosis 
techniques fail. Moreover, the suggested algorithms are able to maintain a comparable 
or a slightly better performance in the cases that can be analyzed by both debugging 
strategies. 

The remainder of the paper is organized as follows: Section|2]gives a brief introduc- 
tion to the main notions of ontology debugging. The details of the suggested algorithms 
and their application are presented in Section [3] In Section |4] we provide evaluation 
results and conclude in Section|5] 

2 Ontology debugging 

Let us exemplify the ontology debugging process by the following use case: 



Example 1. Consider an ontology O with the terminology T: 

ax i : A C B ax 2 : B Q E ax 3 : B C D n ^3s.C 

8i 4 :CCn(J}UE) ax 5 :D\ = ^B 

and assertions .4 : A(i>), s(u, u>)}. Because of the axioms ax 3 and 00:5 the 

given terminology is incoherent and it includes two unsatisfiable classes A and B, 
Moreover, the assertions A(w) and A(v) make the ontology inconsistent. 

Assume that the user is sure that the assertional axioms are correct, i.e. this part of 
the ontology is included to the background knowledge B of the debugging system and, 
thus, cannot be considered as faulty in the debugging process. In this case, the debugger 
can identify the only irreducible set of axioms CS' : ( ax 3 , ax 5 ) - minimal conflict set - 
preserving both inconsistency and incoherency of O. Modification of at least one axiom 
of one of the minimal (irreducible) diagnoses T>[ : [0x3] or T>' 2 : [0x5] is required in 
order to restore both consistency and coherency of the ontology. 

In some debugging systems, e.g. [18], the user can also provide positive P and 
negative N test cases, where each test case is a set of axioms that should be entailed 
(positive) and not entailed (negative) by the ontology resulting in application of the 
debugger. If in the example the user specifies 

P = {{B(w)}} and N = {{^C(w)}} 

then the debugger returns another set of minimal conflicts sets: 

CSi : (axi, ax 3 ) CS2 '■ (ax 2 , 0X4) CS 3 : (ax 3 , ax 5 ) CS4 : (ax 3 , 0x4) 
and diagnoses: 

T>i : [ax3, axi\ T>2 '■ [o>X2, ax 3 ] T> 3 : [ax\, ax^, ax$] 

The reason is that both ontologies 0[ = O \ !)[ and 0' 2 = 0\T>' 2 resulting in applica- 
tion of the diagnoses T)' x and T)' 2 do not fulfill the test cases. For instance, 0' 2 is invalid 
since the axioms {axi, ax 3 } C (D 2 entail -<C{w), which must not be entailed. 

Definition 1 (Target ontology). The target ontology Ot is a set of axioms that is char- 
acterized by a background knowledge B, sets of positive P and negative N test cases. 
The target ontology O t should fulfill the following necessary requirement^ 

- Ot U B must be consistent and, optionally, coherent 

- O t U B |= p VpeP 

- O t UB^n Mne N 

The ontology O is faulty with respect to a predefined B, P and N iff O does not fulfill 
the necessary requirements. 

Definition 2 (Diagnosis problem instance). (0,B, P, N) is a diagnosis problem in- 
stance, where O is a faulty ontology, B is a background theory, P is a set of test cases 
that must be entailed by the target ontology O t and N is a set of test cases that must not 
be entailed by Ot- The instance is diagnosable if B U U peP p is consistent (coherent) 
and B U {J peP p ^= n for each n £ N. 



1 In the following we assume that the user intends to formulate only one ontology. In the paper 
we refer to the intended ontology as the target ontology. 



The ontology debugging approaches [4 8 18] can be applied to any knowledge rep- 
resentation language for which there is a sound and complete procedure for deciding 
whether an ontology entails an axiom or not. Moreover, the entailment relation |= must 
be extensive, monotone and idempotent. 

Another important aspect of the ontology debugging systems comes from the model- 
based diagnosis techniques 1116110 1 that they are based on. The model-based diagnosis 
theory considers the modification operation as a sequence of add/delete operations and 
focuses only on deletion. That is, if an ontology includes faulty axiom then the simplest 
way to remove the fault is to remove the axiom. However, removing an axiom might be 
a too coarse modification, since the ontology can lose some of the entailments that must 
be preserved. Therefore, the model-based diagnosis takes also into account axioms (on- 
tology extension) EX, which are added by the user to the ontology after removing all 
axioms of a diagnosis. Usually the set of axioms EX is either formulated by the user 
or generated by a learning system 1111 . If EX is not empty then all axioms axi <E EX 
are added to the set of positive test cases P, since each axiom axi must be entailed by 
the intended ontology O t . 

Definition 3 (Diagnosis). For a diagnosis problem instance (O, B, P, N) a subset of 
the ontology axioms T> C O is a diagnosis iff Of = (O \ T>) fulfills the requirements of 
Definition^ 

Due to computational complexity of the diagnosis problem, in practice the set of all 
diagnoses is approximated by the set of minimal diagnoses. For a diagnosis problem 
instance (O, B, P, N) a diagnosis T> is minimal iff there is no diagnosis V of the same 
instance such that V C T>. 

The computation of minimal diagnoses in model-based approaches is done by means 
of conflict sets, which are used to constrain the search space. 

Definition 4 (Conflict set). For a problem instance (O, B, P, N) a set of axiom CSC 
O is a conflict set iff one of the conditions holds: 

— CS U B U Up e p is inconsistent (incoherent) or 

- 3n e N such that CS U6U \J peP \= n 

Just as for diagnoses, computation of conflict sets is reduced to computation of minimal 
conflict sets. A conflict set CS is minimal iff there is no conflict set CS 1 such that 

CS' c CS. 

Computation of minimal conflict sets. In practice, the diagnosis systems use two types 
of strategies for computation of conflict sets, namely, brute-force [6 8| and divide-and- 
conquer (7). The first strategy can be split into acquisition and minimization stages. 
During the acquisition stage the algorithm adds axioms of an ontology O \ B to a buffer 
while a set of axioms in the buffer is not a conflict set. As soon as at least one conflict 
set is added to the buffer, the algorithm switches to the minimization stage. In this stage 
axioms are removed from the buffer such that the set of axioms in the buffer remains 
a conflict set after each deletion. The algorithm outputs a minimal conflict set or 'no 
conflicts'. In the worst case a brute force algorithm requires 0(m) calls to the reasoner, 
where m is the number of axioms in a faulty ontology. The algorithm implementing 



divide-and-conquer strategy starts with a buffer containing all axioms of an ontology, 
i.e. the conflict set is in the buffer, and splits it into smaller and simpler sub-problems. 
The algorithm continues splitting until it identifies a sequence of sub-problems includ- 
ing only one axiom such that a set including all these axioms is a minimal conflict set. 
The divide-and-conquer algorithm requires in the worst case 0(k\og(jr)) calls to the 
reasoner, where k is the cardinality of a returned conflict set. Taking into account that 
in practice k <C m, the divide-and-conquer strategy is preferred to the brute-force. 

Identification of minimal diagnoses. The computation of minimal diagnoses in modern 
ontology debugging systems is implemented using the Reiter's Hitting Set HS-Tree 
algorithm 116131 . The algorithm constructs a directed tree from root to the leaves, where 
each node nd is labeled either with a minimal conflict set CS(nd) or / (consistent) or 
x (pruned). The latter two labels indicate that the node is closed. Each edge outgoing 
from the open node nd is labeled with an element s e CS(nd). HS(nd) is a set of edge 
labels on the path from the root to the node nd. Initially the algorithm creates an empty 
root node and adds it to the queue, thus, implementing a breadth-first search strategy. 
Until the queue is empty, the algorithm retrieves the first node nd from the queue and 
labels it with either: 

1. x if there is a node nd' , labeled with either / or x, such that H(nd!) C H(nd) 
(pruning non-minimal paths), or 

2. CS(nd') if a node nd' exists such that its label CS(nd') H H(nd) — (reuse), or 

3. CS if CS is a minimal conflict set computed for the diagnosis problem instance 
(O \ H(nd),B, P, N) by one of the algorithms mentioned above (compute), or 

4. / (consistent). 

The leaf nodes of a complete tree are either pruned (x ) or consistent (/) nodes. The set 
of labels H(nd) of each consistent node nd corresponds to a minimal diagnosis. The 
minimality of the diagnoses is guaranteed due to the minimality of conflict sets, prun- 
ing rule and breadth-first search strategy. Moreover, because of the latter the minimal 
diagnoses are generated in order of increasing cardinality. 

Diagnoses discrimination. In many real-world scenarios an ontology debugger can re- 
turn a large number of diagnoses, thus, placing the burden of diagnosis discrimination 
on the user. Without an adequate tool support the user is often unable to understand the 
difference between the minimal diagnoses and to select an appropriate one. The diag- 
nosis discrimination method suggested in [18] uses the fact that different ontologies, 
e.g. 0\ =0\D\ and O2 — O \ T>2, resulting in the application of different diagnoses, 
entail different sets of axioms. Consequently, there exists a set of axioms Q such that 
Ci |= Q and 02 \£ Q- If such a set of axioms Q exists, it can be used as a query to 
some oracle such as the user or an information extraction system. If the oracle answers 
yes then the target ontology Ot should entail Q and, hence, Q should be added to the 
set of positive test cases P U {Q}. Given the answer no the set of axioms is added to the 
negative test cases NU{Q} to ensure that the target ontology does not entail Q. Thus, in 
the first case the set of axioms T>% can be removed from the set of diagnoses D because 
T>2 is not a diagnosis of the updated diagnosis problem instance (O, B,PU {Q} , N) 
according to Definition [3] Similarly, in the second case the set of axioms T>\ is not a 
diagnosis of (O, B,P,NU {Q}). 



However, many different queries might exist for the set of diagnoses |D| > 2. In 
the extreme case there are 2" — 2 possible queries for a set of diagnoses including n 
elements. To select the best query the authors in lfl8l suggest two measures: SPLIT- 
IN-HALF and ENTROPY. The first measure is a greedy approach preferring the queries 
which allow to remove a half of the minimal diagnoses from D, given an answer of 
an oracle. The second is an information-theoretic measure, which estimates the infor- 
mation gain for both outcomes of each query and returns the one that maximizes the 
information gain. The prior fault probabilities required for ENTROPY measure can be 
obtained from statistics of previous diagnosis sessions. For instance, if the user has 
problems with understanding of restrictions then the diagnosis logs will contain more 
repairs of axioms including restrictions. Consequently, the prior fault probabilities of 
axioms including restrictions should be higher. Given the fault probabilities of axioms, 
one can calculate prior fault probabilities of minimal diagnoses including these axioms 
as well as evaluate ENTROPY (see [18] for more details). 

A general algorithm of the interactive ontology diagnosis process can be described 
as follows: 

1. Generate a set of diagnoses D including at most n diagnoses. 

2. Compute a set of queries and select the best one according to some predefined 
measure. 

3. Ask the oracle and, depending on the answer, add the query either to P or to N. 

4. Update the set of diagnoses D and remove the ones that do not comply with the 
newly acquired test case, according to the Definition [3] 

5. Update the tree and repeat from Step 1 if the queue contains open nodes. 

6. Return the set of diagnoses D. 

The resulting set of diagnoses D includes only diagnoses that are not differentiable 
in terms of their entailments, but have some syntactical differences. The preferred di- 
agnosis in this case should be selected by the user using some text differencing and 
comparison tool. 

Note, that a similar idea can be found in 1 14 | where authors use queries to an oracle 
to revise an ontology. Given a consistent and coherent ontology O the system partitions 
it into two ontologies 0' = and containing required and incorrect consequences cor- 
respondingly. The system can deal with inconsistent/incoherent ontologies if a union of 
all minimal conflict sets is put to the initial set of incorrect consequences Oq . Compu- 
tation of the set requires application of an ontology debugger and is not addressed 
in 111. 

Example 1, continued. Assume that the user mistakenly negated only the restriction in 
a,X3 instead of the whole description B C -n(D\l3s.C). Moreover, in ax4 a disjunction 
was placed instead of conjunction because of a typo, i.e. C C -<(D n E). 

The interactive diagnosis process, illustrated in Fig.[T] applies the three techniques 
described above to find the target diagnosis T> t — [ax^, 0x4]. In the first iteration the 
system starts with the root node, which is labeled with (axi, (1x3) - the first conflict 
returned by QuickXPlain. Next, HS-Tree generates successor nodes and labels the 
edges leading to these nodes with corresponding axioms. The algorithm extends the 
search tree until two leading minimal diagnoses are computed. Given the set of minimal 
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Minimal diagnoses: 

T>i = [0x3, 0x4] 
T>2 = [0x2, 0x3] 

Query: |= {E(w)}7 
Answer: yes 
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Iteration 2 



Minimal diagnoses: 

T>\ — [ax 3 , axi] 
T> 3 = [ax 1, 11x4, axs] 

Query: |= {B(v)}7 
Answer: yes 

No further minimal 
diagnoses, return 

D = {[axi, CIX3]} 



Fig. 1. Identification of the target diagnosis [ax 4, 00:3] using diagnosis discrimination presented 
in 178 1 (HS-Tree and QuickXPlain computing conflicts on-demand). On each iteration two diag- 
noses (n — 2) were computed to identify a query. The answer was used to prune the search tree. 
All computed labels are denoted with C and all reused with 7?. 



diagnoses the diagnosis discrimination algorithm identifies a query using entailments 
of the two ontologies 0\ = 0\T>\ and C 2 = O \ T> 2 , which are deduced by the 
classification and realization services of a standard Description Logic reasoner. One of 
the entailments E(w) can be used as a query, since 0\ |= E(w) and O2 E(w). 
Given a positive answer of an oracle the algorithm updates the search tree and closes 
the node corresponding to the invalid minimal diagnosis T>2- Since there are some open 
nodes, the algorithm continues and finds the next diagnosis U3. The two more nodes, 
expanded by the HS-Tree in the second iteration, are closed since both sets of labels 
on the paths to these nodes from the root are supersets of the closed paths {ax^, ax^\ 
and { 02:3, CLX4}. For the two minimal diagnoses T>i and 2?3 the diagnosis discrimination 
finds a query Q — {B(v)}, which is answered positively by the oracle. Consequently, 
the algorithm removes T>^ and continues with the expansion of the last node labeled 
with (axz, ax 4 ). Since the paths to the successors of this node are supersets of existing 
closed paths in the tree, the algorithm closes these nodes and terminates. The diagnosis 
T>i suggesting modification of the axioms ax^ and ax^ is returned to the user. 

The example shows that a modern ontology debugger can efficiently identify the 
target diagnosis. As it is demonstrated by different evaluation studies 1181 1 81 . the debug- 
gers work well in an ontology development and maintenance process in which users 
modify an ontology manually. In such process the users classify ontology regularly 
and, therefore, are able to identify the presence of faults early, i.e. a user introduces 
only a small number of modifications to the ontology between two calls to a reasoner. 
Therefore, faulty ontologies in this scenario can be characterized by a small number 



of minimal conflict sets that can generate a large number of possible diagnoses. For 
instance, Transportation ontology (see [18]) includes only 9 minimal conflict sets that 
generate 1782 minimal diagnoses. In such case HS-Tree makes only 9 calls to QuiCK- 
XPlain and then reuses the identified minimal conflicts to label all other nodes. The 
number of calls to the reasoner, which is the main "source of complexity", is rather low 
and can be approximated by 9fclog ? + [Nodes], where | Nodes | is the cardinality of 
the set containing all nodes of the search tree, k is the maximum cardinality of all com- 
puted minimal conflict sets and n is the number of axiom in the faulty ontology. The 
combination of ontology debugging with diagnosis discrimination allows to reduce the 
number of calls to the reasoner, because often acquired test cases invalidate not only 
diagnoses that are already computed by HS-Tree, but also those that are not. All these 
factors together with such techniques as module extraction [17] make the application of 
ontology debuggers feasible in the described scenario. 

However, in such applications as ontology matching or learning the number of min- 
imal conflict sets can be much higher, because all axioms are generated at once. For in- 
stance, ontology alignments, identified by most of ontology matching systems in the last 
Ontology Alignment Evaluation Initiative (OAEI), are often incoherent and, in some 
cases, inconsistent Qj. The large number of minimal conflict set makes the application 
of ontology debugging problematic because of the large number of calls to the reasoner 
and the memory required by the breadth-first search algorithm. To overcome this prob- 
lem we suggest a novel ontology debugging approach that computes diagnoses directly, 
i.e. without precomputation of minimal conflict sets. 

3 Direct diagnosis of ontologies 

The main idea behind the approach is to start with the set T>o = and extend it until 
such a subset of ontology axioms T> C O is found that T> is a minimal diagnosis with 
respect to the Definition[3] In the first step Algorithm[T]verifies the input data, i.e. if the 
input ontology O, background knowledge B, positive P and negative N test cases to- 
gether constitute a valid diagnosis problem instance (O, B, P, N) (Definition]^. Thus 
it verifies: a) whether the background theory together with the positive and negative test 
cases is consistent; and b) if the ontology is faulty. In both cases Inv-QuickXPlain 
calls VERIFYREQUIREMENTS function that implements Definition [5] and tests if ax- 
ioms in the set I? are a minimal diagnosis or not. The test function requires a reasoner 
that implements consistency/coherency checking (isConsistent) and allows to de- 
cide whether a set of axioms is entailed by the ontology (ENTAILS). 

FINDDIAGNOSIS is the main function of algorithm which takes six arguments as in- 
put. The values of the arguments B, O and N remain constant during the recursion and 
are required only for verification of requirements. Whereas values of T>, A and O a are 
used to provide a set of axioms corresponding to the actual diagnosis and two diagnosis 
sub-problems on the next level of the recursion. The sub-problems are constructed dur- 
ing the execution FINDDIAGNOSIS by splitting a given diagnosis problem with SPLIT 
function. In the most of the implementations SPLIT simply partitions the set of axioms 
into two sets of equal cardinality. The algorithm continues to divide diagnosis problems 
(FlNDDlAGNOSES line 12) until it identifies that the set V is a diagnosis (line 7). In fur- 
ther iterations the algorithm minimizes the diagnosis by splitting it into sub-problems 



Algorithm 1: Inv-QuickXPlain(C, B, P, N) 



Input: O set of faulty axioms, B set of background axioms, P set of positive test cases, N 

set of negative test cases 
Output: a minimal diagnosis 23 
iO'<-0\S; 

2 6VBuU peP p; 

3 if ^VERIFYReouirements(,B',0,0, N) then return 'inconsistent requirements'; 

4 if verifyRequirements (23', 0, C, N) then return 'consistent'; 
s return findDiagnosis(S', 0, 0', O', N); 

6 function findDiagnosis (B, 23, A, Oa, O, N) returns a minimal diagnosis 23 

7 if A ^ A verifyRequirements(Z3, 23, O, N) then return 0; 

8 if\0\ = 1 then return©; 

9 k <s— SPLIT(O); 

io Ci «- getElements (Oa , 1, fc); 

n C 2 <— getElements (Oa , fc — 1, | | ) ; 

12 T> 2 <r- findDiagnosis(B,Ou Oi,Oi,0 2 ,0,N); 

13 V 1 <r- findDiagnosis (B, 23 UI»2,I»2, Oi,Q,N); 

14 return T>\ U 23 2 ; 

is function verifyRequirements (Z3, 23, 0, N) returns true ox false 

w 0VBU(0\C); 

n c isConsistent(O') ; 

is if then return false; 

19 foreach n e N do c -h- cA entails (C, n); 

20 return c ; 



of the form T> = V U O^, where contains only one axiom. In the case when T> is 
a diagnosis and V is not, the algorithm decides that Oa is a subset of the sought min- 
imal diagnosis. Just as the original algorithm, Inv-QuickXPlain always terminates 
and returns a minimal diagnosis for a given diagnosis problem instance. 

Example^ continued. Let us look again at the ontology diagnosis example and show 
how a diagnosis is computed by Inv-QuickXPlain (see Pig. |2j. The algorithm starts 
with an empty diagnosis T> = and Oa containing all axioms of the problem. VER- 
IFYREQUIREMENTS returns false since the B U O \ is inconsistent. Therefore, the 
algorithm splits Oa into {axi, ax-i) and {ax^, ax4, ax^} and passes the sub-problem 
to the next level of recursion. Since, the set 23 = {axi, ax 2 } is not a diagnosis, the 
ontology B U (O \ 23) is inconsistent and the problem in Oa is split one more time. On 
the second level of recursion the set 23 is a diagnosis, although not minimal. The func- 
tion verifyRequirements returns true and the algorithm stalls to analyze the found 
diagnosis. Therefore, it verifies whether the last extension of the set 23 is a subset of a 
minimal diagnosis. Since, the extension includes only one axiom ax^ an the extended 
set {ax i, 0x2} is not a diagnosis, the algorithm concludes that ax^ is an element of 
the target diagnosis. The left-most branch of the recursion tree terminates and returns 
{02:3}. This axiom is added to the set 23 and the algorithm starts investigating whether 
the two axioms {axi, 0x2} also belong to a minimal diagnosis. First, it tests the set 



V = 
A = 

C A = {ax 1 ,ax 2 ,ax 3 ,ax i , ax 5 }^ 
verifyRequirements 



— > return {ax 3 , ax 2 } 



{ax 3 , ax 2 ] U ( 



2? = {ax lt ax 2 ] 
A = {ax 1( ax 2 } 
£>A = {ax 3 , ax 4 , ax 5 ] 
verifyRequirements ^ 



{ax 3 } U . 



P = {ax 1; ax 2 , ax 3 ] 
A = {ax 3 ) 
A = {ax 4 , ax 5 } 
verifyRequirements S 




= {axj, ax 2 ) 
A = 
O a = {ax 3 } 
verifyRequirements^ 



£> = {ax 3 } 
A = {ax 3 } 

verifyRequirements-^ 





D = {axs.axj) 
A = {axj) 
Oa = {a* 2 } 

VERIFYREQUIREMENTS^- 



V = {ax 3 , ax 2 } 
A = {ax 2 } 
O a = {axj} 
verifyRequirements^ 



Fig. 2. Recursive calls of Inv-HS-Tree for the diagnosis problem instance in Example [T] The 
background theory B, original ontology O and the set of negative test cases N remain constant 
and therefore are omitted. Solid arrows show recursive calls (line 12 - left and line 13 - right) 
and dashed arrows indicate returns. 



{ax3, axi}, which is not a diagnosis, and on the next iteration it identifies the correct 
result {axs, ao^}- 

Inv-QuickXPlain is a deterministic algorithm and returns the same minimal di- 
agnosis if applied twice to a diagnosis problem instance. In order to obtain different 
diagnoses, the problem instance should be changed such that the Inv-QuickXPlain 
will identify the next diagnosis. Therefore, we suggest Inv-HS-Tree, which is a modi- 
fication of the HS-Tree algorithm presented in Section[2] The inverse algorithm labels 
each node nd of the tree with a minimal diagnosis T>(nd). The rules 1, 2 and 4 of 
the original algorithm remain the same in Inv-HS-Tree and the rule 3 is modified as 
follows: 

3. The open node nd is labeled with T> if T> is a minimal diagnosis for the diagnosis 
problem instance (O, B U H(nd), P, N) computed by Inv-QuickXPlain (com- 
pute) 

where H(nd) is a set containing edge labels on the path from the root to nd. In this case 
elements of H{nd) correspond to the axioms of minimal diagnoses that were used as 
labels of nodes on the path. Addition of an axiom aXk of a minimal diagnosis 2?j to the 
background theory forces Inv-QuickXPlain to search for a minimal diagnosis that 
suggests the modification of any other axiom, except axk - That is, the diagnosis T>i will 
not be rediscovered by the direct diagnosis algorithm. 

A modified update procedure is another important feature of Inv-HS-Tree. In the 
diagnosis discrimination settings the ontology debugger acquires new knowledge that 
can invalidate some of the diagnoses that are used as labels of the tree nodes. During 
the tree update Inv-HS-Tree searches for the nodes with invalid labels. Given such a 
node, the algorithm removes its label and places it to the list of open nodes. Moreover, 
the algorithm removes all the nodes of a subtree originating from this node. After all 



[ax2, 0X3] 



Diagnoses: 

T>\ = [ax2, (1x3] 

T>2 = [0X3, CIX4] 




Diagnoses: 

V2 = [ax-i, axi\ 
T>3 = [axi, axi, ax$] 




Query: |= E(w) 
Answer: yes 



[ax 1, axA, axs] 



Query: |= B(v) 
Answer: yes 



No further minimal 
diagnoses, return 
D = {[03:3, aau]} 



Fig. 3. Identification of the target diagnosis [0x3, ax^] using direct diagnosis approach. 

nodes with invalid labels are cleaned-up, the algorithm attempts to reconstruct the tree 
by reusing the remaining valid minimal diagnoses (rule 2, HS-Tree). Such aggres- 
sive pruning of the tree is feasible since a) the tree never contains more than n nodes 
that were computed with Inv-QuickXPlain and b) computation of a possible mod- 
ification to the minimal diagnosis, that can restore its validness, requires invocation of 
Inv-QuickXPlain and, therefore, as hard as computation of a new diagnosis. Note 
also, that in a common diagnosis discrimination setting n is often set to a small num- 
ber, e.g. 10, in order to achieve good responsiveness of the system. Consequently, in 
this settings the size of the tree will be small. The latter is another advantage of the 
direct method as it requires much less memory in comparison to a debugger based on 
the breadth-first strategy. 

Example [7] continued. Applied to the sample diagnosis problem instance the direct 
ontology debugger computes two minimal diagnoses [ax 2 , ax 3 ] and [ax 3 , 0x4] in the 
first iteration (see Fig. [3]). For these diagnoses the discrimination method identifies the 
query E(w), which is answered yes by the oracle. The label of the root node in this case 
becomes invalid. Consequently, the algorithm removes the label of the root, deletes its 
subtree and places the root to the list of the open nodes. Next, according to the rule 2, 
the valid minimal diagnosis T> 2 is reused to label the root. Given a diagnosis problem 
instance (O, B U {ax 3} ,P,N), Inv-QuickXPlain computes the last minimal diag- 
nosis U3. Given the positive answer to the query B{v) the algorithm labels both open 
nodes with / and returns T>2 as the result. Moreover, the labels on the edges of the 
tree correspond to the minimal conflicts (ax 3 ) and (ax±) of the final diagnosis problem 
instance. 

4 Evaluation 

We evaluated the direct ontology debugging technique using aligned ontologies gen- 
erated in the framework of OAEI 2011 (TJ. These ontologies represent a real- world 
scenario in which a user generated ontology alignments by means of some (semi- 
automatic tools. In such case the size of the minimal conflict sets and their configu- 
ration might be substantially different from the ones considered in the manual ontology 
development process, e.g. |8 18 1. In the first experiment we demonstrate that Inv-HS- 
Tree is able to identify minimal diagnoses in the cases when HS-Tree fails. The 




Fig. 4. Time required to compute 1, 9 and 30 diagnoses by HS-TREE and Inv-HS-Tree for the 
Conference problem. 



second test shows that the direct diagnosis approach is scalable and can be applied to 
ontologies including thousands of axioms. 

The ontology matching problem can be formulated as follows: given two ontologies 
Oi and Oj, the goal of the ontology matching system is to identify a set of alignments 
My. Each element of this set is a tuple (a^, Xj, r, v), where xi € Q(Oi), Xj € Q(Oj), 
r is a semantic relation and v is a confidence value. Q(0) denotes a set of all matchable 
elements of an ontology O such as classes or properties. The result of the ontology 
matching process is the aligned ontology 0y = Oi U My U Oj . In the ontologies, used 
in both experiments, only classes and properties were considered as matchable elements 
and the set of relations was limited tor e {C, =, □}. 

In the first experiment we applied the debugging technique to the set of aligned on- 
tologies resulting from "Conference" set of problems, which is characterized by lower 
precision and recall of the applied systems (the best F-measure 0.65) in comparison, 
for instance, to the "Anatomy" problem (average F-measure 0.8^] The Conference test 
suite^] includes 286 ontology alignments generated by the 14 ontology matching sys- 
tems. We tested all the ontologies of the suite and found that: a) 140 ontologies are 
consistent and coherent; b) 122 ontologies are incoherent; c) 26 ontologies are incon- 



2 see http://oaei.ontologymatching.org/201L5/results/index.html for preliminary results of the 
evaluation based on reference alignments 

3 All ontologies used in the evaluation can be downloaded from 
http://code.google.eom/p/rmbd/wiki/DirectDiagnosis The webpage contains also tables 
presenting detailed results of the experiment presented in Fig.|4] 
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Table 1. Ontologies diagnosable only with Inv-HS-Tree. 



sistent; and in 8 cases HermiT Tl3l was unable to finish the classification in two hour^J 
The results show that only two systems CODI and MaasMtch out of 14 were able to 
generate consistent and coherent alignments. This observation confirms the importance 
of high-performance ontology debugging methods. 

The 146 ontologies of the cases b) and c) were analyzed with both HS-Tree and 
Inv-HS-Tree. For each of the ontologies the system computed 1, 9 and 30 leading 
minimal diagnoses. The results of the experiment presented in Fig|4]show that for 133 
ontologies out of 146 both approaches were able to compute the required amount of 
diagnoses. In the experiment where only 1 diagnosis was requested, the direct approach 
outperforms the HS-Tree as it was expected. In the next two experiments the time 
difference between the approaches decreases. However, the direct approach was able to 
avoid a rapid increase of computation time for very hard cases. 

In the 13 cases presented in Table[T]the HS-Tree was unable to find all requested 
diagnoses in each experiment. Within 2 hours the algorithm calculated only 1 diagnosis 
for csa-conf erence-ekaw and for ldoa-conf erence-conf of it was able 
to find 1 and 9 diagnoses. The results of the Inv-HS-Tree are comparable with the 
presented in the Fig. [4] This experiment shows that the direct diagnosis is a stable and 
practically applicable method even in the cases when an ontology matching system 
outputs results of only moderate quality. 

Moreover, in the first experiment we evaluated the efficiency of the interactive di- 
rect debugging approach applied to the cases listed in Table [T] In order to select the 
target diagnosis we searched for all possible minimal diagnoses of the following diag- 
nosis problem instance (Mf, Oi U Oj U M t , 0, 0), where Mf and M t are the sets of 
false and true positive alignments. Both sets can be computed from the set of correct 
alignments M c , provided by the organizers of OAEI 2011, and the set My generated 
by a ontology matching system as Mf = My \ M c and M t = Mij n M c . From this 

4 The tests were executed on a Core-i7 (3930K) 3.2Ghz, 32GB RAM and with Ubuntu Server 
1 1.04, Java 6 and HetmiT 1.3.6 installed. 
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Table 2. Diagnosis discrimination using direct ontology debugging. Scoring stands for query se- 
lection strategy, react system reaction time between queries, #CC number of consistency checks, 
CC gives average time needed for one consistency check. 



set of diagnoses we choose one diagnosis at random as the target. In the experiment the 
prior fault probabilities of diagnoses were assumed to be 1 — v, where v is the confi- 
dence value of the ontology matching system that the alignment is correct. Moreover, 
all axioms of both ontologies Oi and Oj were assumed to be correct and were assigned 
small probabilities. 

The results presented in Table [2] were computed using both split-in-half (SPL) 
and entropy measure (ENT) for query selection for the diagnosis problem instance 
(Mij,Oi U Oj, 0, 0). The entropy measure was able to to solve the problem more effi- 
ciently because it is able to use information provided by the ontology matcher in terms 
on confidence values. The experiment shows also that efficiency any debugging meth- 
ods depends highly on the ability of the underlying reasoner to classify an ontology. 
Note that the comparison of the suggested debugging technique with the ones build- 
in to such ontology matching systems as CODI [ 15 1 or LogMap is inappropriate, 
since all these systems use greedy diagnosis techniques (e.g. lfT2lO . whereas the method 
presented in this paper is complete. However, the results presented in Table [T] as well 
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Table 3. Scalability test for Inv-HS-Tree, time given in seconds. 



as in Fig. [4] indicate that the suggested approach can find one minimal diagnosis in 25 
seconds on average, which is comparable with the time of the mentioned systems. 

In the second evaluation scenario we applied the direct method to unsatisfiable on- 
tologies, generated for the Anatomy problem. The source ontologies 0\ and O2 include 
1 1545 and 4838 axioms correspondingly, whereas the size of the alignments varies be- 
tween 1147 and 1461 axioms. The diagnosis selection process was performed in the 
same way as in the first experiment, i.e. we selected randomly one of the diagnoses of 
the instance (Mf, 0\ U O2 U M t , 0, 0). The tests were performed for the problem in- 
stance diagnosis (M12, 0\ U O2, 0, 0) for 7 of the 12 systems. We excluded the results 
of CODI, CSA, MaasMtch, MapEVO and Aroma, because CODI produced coherent 
alignments and the output of the other systems was not classifiable within 2 hours. The 
results of the experiment show that the target diagnosis can be computed within 40 
second in an average case. Moreover, Inv-HS-Tree slightly outperformed HS-Tree. 



5 Conclusions 

In this paper we present an approach to direct computation of diagnoses for ontology 
debugging. By avoiding computation of conflict sets, the algorithms suggested in the 
paper are able to diagnose the ontologies for which a common model-based diagnosis 
technique fails. Moreover, we show that the method can also be used with diagnosis 
discrimination algorithms, thus, allowing interactive ontology debugging. The experi- 
mental results presented in the paper indicate that the performance of a system using the 
direct computation of diagnoses is either comparable with or outperforms the existing 
approach in the settings, when faulty ontologies are generated by ontology matching 
or learning systems. The scalability of the algorithms was demonstrated on a set of big 
ontologies including thousands of axioms. 
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